Predictive instruction text with virtual lab representation highlighting

ABSTRACT

A lab automation system receives an instruction from a user to perform a protocol within a lab via an interface including a graphical representation of the lab. The lab includes a robot and set of equipment rendered within the graphical representation of the lab. The lab automation system identifies an ambiguous term of the instruction and pieces of equipment corresponding to the ambiguous term and modifies the interface to include a predictive text interface element listing the pieces of equipment. Upon a mouseover of a listed piece of equipment within the predictive text interface element, the lab automation system modifies the graphical representation of the lab to highlight the listed piece of equipment corresponding to the mouseover. Upon a selection of the listed piece of equipment within the predictive text interface element, the lab automation system modifies the instruction to include the listed piece of equipment.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.63/060,974 filed Aug. 4, 2020, which is incorporated by reference in itsentirety.

BACKGROUND

In traditional lab environments, human operators work throughout the labto perform protocols with equipment and reagents. For example, a humanoperator may mix reagents together, manually calibrate a robot arm, andoperate a pipettor robot to handle liquids. However, in some instances,a lab may include components (e.g., equipment, robots, etc.) that can beautomated to perform protocols.

Though automating protocols may streamline the necessary processes,automation in lab environments poses unique challenges. For one, thelanguage used by operators or for robots and equipment in labs is notstandardized, so communications about protocols for a lab system toperform may be difficult for the lab system to correctly parse.Secondly, operators in labs may not be versed in how to use a lab systemfor automation given their specific scientific backgrounds. Further,although some of robots and equipment may be capable of easy integrationinto the lab system for automation, not all robots or equipment may beconfigured for automation and may lack the appropriate interfaces forthe lab system to communicate with. Lastly, each of a range of robotsand equipment connected to the lab system may have its own interface forcommunicating, and the lab system may need to determine how tocommunicate with each different interface, which may increase latency.

SUMMARY

The following disclosure describes a lab automation system that performsprotocols in a lab. In particular, the lab automation system modifiesgraphical representations of labs based on instructions and interactionsreceived via a graphic user interface.

A lab automation system operates in a lab environment using componentssuch as equipment, reagents, and robots to perform protocols. The labautomation system renders graphical representations of labs. The labautomation system may receive instructions from a user (sent via agraphic user interface) indicating a protocol to be performed in a laband identifies ambiguous terms in the instructions. The lab automationsystem modifies the graphic user interface to include predictive testinterface elements based on the ambiguous terms such that the user mayinteract with the graphic user interface to select equipment associatedwith the ambiguous terms.

In particular, the lab automation system receives an instruction from auser to perform a protocol within a lab via an interface including agraphical representation of the lab. The lab includes a robot and set ofequipment rendered within the graphical representation of the lab. Thelab automation system identifies an ambiguous term of the instructionand one or more pieces of equipment corresponding to the ambiguous termand modifies the interface to include a predictive text interfaceelement listing the one or more pieces of equipment corresponding to theambiguous term. Upon a mouseover of a listed piece of equipment withinthe predictive text interface element, the lab automation systemmodifies the graphical representation of the lab to highlight the listedpiece of equipment corresponding to the mouseover. Upon a selection ofthe listed piece of equipment within the predictive text interfaceelement, the lab automation system modifies the instruction to includethe listed piece of equipment.

The features and advantages described in the specification are not allinclusive and, in particular, many additional features and advantageswill be apparent to one of ordinary skill in the art in view of thedrawings, specification, and claims. Moreover, it should be noted thatthe language used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system environment for a lab automation system,according to one embodiment.

FIG. 2A illustrates components of a lab, according to one embodiment.

FIG. 2B illustrates an example robot configured to perform protocols ina laboratory environment, according to one embodiment.

FIG. 2C illustrates a lab, according to one embodiment.

FIG. 3 illustrates a high-level block diagram of the lab automationsystem, according to one embodiment.

FIG. 4 illustrates a graphic user interface for the lab automationsystem, according to one embodiment.

FIG. 5A illustrates an example graphic user interface for inputtinginstructions, according to one embodiment.

FIG. 5B illustrates an example graphic user interface highlightingequipment in a virtual representation of a lab, according to oneembodiment.

FIG. 6A illustrates an example graphic user interface depicting aselection of a virtual element within a virtual representation of a lab,according to one embodiment.

FIG. 6B illustrates an example graphic user interface with a tagelement, according to one embodiment.

FIG. 7 illustrates an example user interface depicting a simulation,according to one embodiment.

FIG. 8 illustrates a process for configuring a robot to perform stepsfor a protocol, according to one embodiment.

FIG. 9 illustrates a process for simulating a protocol in a virtualrepresentation of a lab, according to one embodiment.

FIG. 10 illustrates a process for modifying a virtual representation,according to one embodiment.

FIG. 11 illustrates a process for determining a location of labequipment relative to a robot arm, according to one embodiment.

FIG. 12 illustrates a process for configuring a second robot, accordingto one embodiment.

FIG. 13 illustrates a process for modifying an instruction uponselection of a piece of equipment, according to one embodiment.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION System Overview

FIG. 1 illustrates a system environment for a lab automation system 100,according to one embodiment. The lab automation system 100 is connectedto a number of client devices 120 used by operators of one or more labs140 via a network 110. These various elements are now described inadditional detail.

The client devices 120 are computing devices such as smart phones,laptop computers, desktop computers, or any other device that cancommunicate with the lab automation system 100 via the network 110. Theclient devices 120 may provide a number of applications, which mayrequire user authentication before a user can use the applications, andthe client devices 120 may interact with the lab automation system 100via an application. The client devices may present graphic userinterfaces displaying information transmitted from the lab automationsystem 100. Though two client devices 120 are shown in FIG. 1, anynumber of client devices 120 may be connected to the lab automationsystem 100 in other embodiments. The client devices 120 may be locatedwithin labs 140 connected to the lab automation system 100 or externalto the labs.

The network 110 connects the client devices 120 to the lab automationsystem 100, which is further described in relation to FIG. 3. Thenetwork 110 may be any suitable communications network for datatransmission. In an embodiment such as that illustrated in FIG. 1, thenetwork 110 uses standard communications technologies and/or protocolsand can include the Internet. In another embodiment, the network 110 usecustom and/or dedicated data communications technologies.

The labs 140 are connected to the lab automation system 100 via thenetwork 110. A lab 140 is a physical space equipped for completingresearch, experiments, or manufacturing of various products. Each lab140 includes one or more of robots 150, a camera system 160, labequipment 170, and reagents 180. The robots 150 may be mobilized tosynthesize products or conduct research and experiments in the lab 140.Examples of robots 150 that may be used in the labs 140 are liquidhandlers, microplate movers, centrifuges, cappers/decappers, sorters,labelers, loaders, and the like. Each robot 150 may include one or moresensors attached to elements of the robots 150, such as positionsensors, inertial measurement units (IMUs), accelerometers, cameras, andthe like. Each robot 150 may also include one or more tags attached toexternal elements of the robot 150. The tags may be visible in imagedata captured of the lab by the camera system 160, described below, suchthat the lab automation system may determine positions of the externalelements of the robots 150 and calibrate cameras of the camera system160 based on the tags.

Each lab 140 includes a camera system 160 comprising one or morecameras. The cameras may be video cameras, infra-red cameras,thermographic cameras, heat signature cameras, or any other suitablecamera. The cameras of a camera system 160 may be interspersedthroughout a lab 140 to capture images and/or video of the lab 140,which may be used by the lab automation system 100 to calibrate therobots 150 and/or lab equipment 170 within the lab.

The lab equipment 170 (or, simply, “equipment”) in the lab is used bythe robots 150 and/or human operators to manufacture products frommaterials (e.g., reagents 180) or conduct experiments/research withinthe lab. Each piece of equipment 170 may be operated by robots 150,human operators, or both. Examples of equipment 170 may includepipettes, beakers, flasks, plates, storage equipment, incubators, platereaders, washers, centrifuges, liquid handlers, sealers, desealers, orany other suitable equipment used in labs 140. The equipment may be usedto synthesize products based on one or more reagents 180 stored in thelabs 140. Reagents 180 are substances that may be mixed together forchemical reactions. Examples of reagents 180 stored in the labs 140 mayinclude acetic acid, acetone, ammonia, ethanol, formaldehyde, hydrogenperoxide, sodium hydroxide, and the like. The lab automation system 140,which is further described in relation to FIG. 3, maintains a record ofthe robots 150, equipment 170, and reagents 180 stored at each lab 140connected to the lab automation system 100.

FIG. 2A illustrates components of a lab 140C, according to oneembodiment. The lab 140C includes a surface 205 and may include one ormore components such as robots 150, equipment 170, and reagents 180. Inparticular, the lab 140C includes a robot 150C programmable to performoperations in the lab 140C (e.g., by interacting with pieces ofequipment 170). In the example of FIG. 2A, the lab 140C additionallyincludes a camera 215 positioned to receive image data describing thelab 140C. In other examples, the camera 215 may be mounted in differentlocations in the environment 200 or may be incorporated in the robot150C or attached to an element 220 of a component.

FIG. 2B illustrates an example robot 150D configured to performprotocols in a lab 140D according to one embodiment. In the embodimentof FIG. 2B, the robot 150D comprises components including a robot arm225 and a robot hand 230. A camera of a camera system 160 in the lab140D enables the lab automation system 100 to receive image datadescribing the lab 140D, including the robot 150D and any pieces ofequipment 170 in the lab 140D. The robot arm 225 comprises one or morejointed, movable pieces with one or more degrees of freedom of motion,and connects a body of the robot 150D to the robot hand 230. The robothand 230 comprises one or more jointed, movable pieces configured tointeract with the components in the lab 140D. For example, the robothand 230 can include a set of digits, pincers, or claws for moving andlifting pieces of equipment 170 such as pipettes, beakers, and the like,and for interacting with pieces of equipment 170, e.g., inputting ormodifying settings, accessing compartments, and the like. In otherembodiments, the robot 150D may include additional, fewer, or differentelements than those shown in FIG. 2B, and the robot arm 225 and/or therobot hand 230 may include different elements than those shown in FIG.2B.

FIG. 2C illustrates a lab 140D, according to one embodiment. The lab140D includes lab equipment 170 and robots 150 with robot arms 225 androbot hands 230. The robots 150 may interact with the lab equipment 170to move reagents and other materials for laboratory protocols conductedin the lab 140D. For instance, the robots 150 may access a reagentstored in lab equipment 170E and transfer the reagent to lab equipment170D using pipettes held by the robot hands 230. Though numerouscomponents are shown in FIG. 2C (e.g., lab equipment such asrefrigerators, cabinets, venting systems, etc. and robots such as liquidhandlers), other labs 140 connected to the lab automation system 100 maycontain other components in other configurations throughout the labs140.

FIG. 3 illustrates a high-level block diagram of the lab automationsystem 100, according to one embodiment. The lab automation system 100includes an instruction module 310, a rendering module 320, a protocolmodule 330, a simulation module 340, a calibration module 350, a graphicuser interface module 355, a representation database 360, one or moremachine learned models 370, a lab database 380, and a protocol database390. In some embodiments, the lab automation system 100 may includeadditional or alternative modules or databases not shown in FIG. 3.

The instruction module 310 determines steps from a set of instructions.The instruction module 310 receives instructions from the graphic userinterface module 355. The instructions indicate how to perform aprotocol in a lab 140 and are a set of text input by a user via agraphic user interface displayed via a client device 120. For example,the text of instructions may indicate to “load holders into Hamiltonliquid handler, load aluminum polymer, and pipette polymer intoholders.”

The instruction module 310 converts the text of the instructions into aset of steps. For instance, the instruction module 310 may identify verbphrases within the instructions that indicate operations that need to bedone for the protocol. The verb phrases may each include a verb and oneor more dependent words that the verb describes, and each verb phrasemay correspond to a step. For example, the instruction may identifythree verb phrases in the text “load holders into Hamilton liquidhandler, load aluminum polymer, and pipette polymer into holders”: “loadholders,” “load aluminum polymer,” and “pipette.” In some embodiments,the instruction module 310 may apply one or more natural languageprocessing methods and/or machine-learned models 370 to determine thesteps of the instructions. The machine learned model 370 may be trainedon example texts labeled with one or more steps. The instruction module310 may store the determined steps in association with the text or acolloquial name of the protocol, which may be received with the textfrom the graphic user interface module 355, in the protocol database390.

The instruction module 310 identifies one or more of an operation, labequipment 170, and reagent 180 for each step of the set of steps. Anoperation may be described by a verb of the verb phrase associated withthe step. The operation may be performed using one or more of labequipment 170, robots 150, and/or reagents 180 (e.g., components), whichmay be a complement phrase or adjunct phrase explicit in the verbphrase. For instance, the lab equipment 170 and reagent 180 may berepresented by one or more nouns following the verb (e.g., “the holders”and “the aluminum polymer” in the step “load the holders with thealuminum polymer.”). In some embodiments, the lab equipment 170 andreagents 180 may be implicit in the step. For instance, the step “loadthe saline solution” may be associated with the robot “Hamilton liquidhandler,” which is used to load reagents in labs 140. In someembodiments, the instruction module 310 may identify the operation, labequipment 170, robot 150, and/or reagent 180 for each step using one ormore natural language processing methods (e.g., creating a syntax tree)and/or machine learned model(s) 370 trained on a set of example stepslabeled with operations, lab equipment 170, and reagents 180. Theinstruction module 310 stores the identified operations, lab equipment170, and reagents 180 with the associated steps in the protocol database390.

In some embodiments, the instruction module 310 may determine a lab 140to perform the protocol in based on the text of the instructions. Insome instances, the text may be associated with a particular lab 140indicated by a user of where to perform the protocol associated with theinstructions. In other instances, the instruction module 310 maydetermine a lab 140 for the protocol to be performed in by parsing thetext to identify a noun (and one or more descriptive adjectives)representing a lab 140 and compare the parsed text to an index in thelab database 380 that associates labs 140 with colloquial names used byoperators, locations of the labs 140, equipment 170, and reagents 180stored at the labs 140, and the like. For example, the instructionmodule 310 may determine the text “prep holders in main street lab” isassociated with “Lab 1 on Main Street” of the lab database 380. Further,the instruction module 310 may select a lab 140 based on parametersreceived with the instructions and additional factors such as schedulesof the labs 140, operator availability at each lab, equipment availableat each lab, reagents available at the lab, and the like. The parametersmay describe a desired protocol location for performing the protocol, adesired proximity to the protocol location, equipment preferences,reagent preferences, budget for the protocol, desired amount ofautomation for the protocol, desired quality level, and the like.

The instruction module 310 may detect one or more ambiguities in theidentified operations, equipment 170, and reagents 180 of the steps. Anambiguity may be a word or phrase of the instructions that theinstruction module 310 is unable to map to a component (not includingprepositions, articles, and conjunctions). To detect ambiguities, theinstruction module 310 may remove articles, conjunctions, andprepositions from the instructions and cross-reference each word withthe lab database 380 and/or an external dictionary. If the instructionmodule 310 is still unable to resolve the ambiguity, the instructionmodule 310 may send an indication to the graphic user interface module355 to present one or more words associated with the ambiguity to a userand receives one or more selected words from the graphic user interfacemodule 355. The instruction module 310 updates the steps in the protocoldatabase 390 based on the selected words.

The instruction module 310 may also detect one or more errors in thesteps. An error may be a combination of one or more components thatcould not practically function together to perform an operation or usingcomponents to perform an operation that would violate a set of safetyprotocols. The instruction module 310 may detect errors using acombination of natural language processing and information about labs140 in the lab database 380. For instance, the instruction module 310may detect the textual segment “move refrigerator with liquid handler”as including an error since the liquid handler is used to move liquidsand could not lift the refrigerator. To resolve an error, theinstruction module 310 may send a list of components associated with astep to the graphic user interface module 355 for display to a user. Theinstruction module 310 receives a selection of one or more componentsfrom the graphic user interface module 355 and updates the step in theprotocol database 390 to use the selected components. In someembodiments, the instruction module 310 may determine one or morevariants for the components or operation and update the protocol withthe variants.

The instruction module 310 may determine a set of variants for thecomponents of each step. Variants are alternate components that may beused in place of the lab equipment 170, robots 150, and reagents 180(henceforth “determined components” for simplicity). Each variant mayhave the same or alternate (e.g., similar) functionality to acorresponding determined component. For instance, variants with the samefunctionality may perform or be used for the exact same operations as acorresponding determined component. For example, a first robot may beable to lift up 20 pounds of material for an operation, and a secondrobot that can lift up to 30 pounds of material would be a variant ofthe first robot. Variants with alternate functionality may perform or beused for substantially similar operations to the determined components.For example, a first reagent may have a specific reaction when mixedwith a second reagent. A variant of the first reagent may be a thirdreagent 180 that has the same specific reaction with the second reagent180 (in instances where no other reagents are mixed with the describedreagents 180). In some embodiments, the instruction module 310 may alsodetermine variants with limited overlapping functionality to thecomponents. In further embodiments, the instruction module 310 maydetermine the variants based on parameters received with theinstructions. The instruction module 310 may account for the parameters(e.g., cost and time to perform the protocol, skillset required of ahuman operator, etc.) in determining variants.

The instruction module 310 may determine the variants by accessing anindex of the lab database 380 to cross-reference components used toperform the same operations. For instance, the instruction module 310may determine that the lab equipment “liquid holder,” “pipette machine,”and “handheld dispenser” may all be used to dispense liquid and relatethe lab equipment 170 as variants of one another in the lab database380. In some embodiments, the instruction module 310 may apply a machinelearned model 370 to the components to determine the variants. Themachine learned model may be trained on components labeled with sets ofvariants. The instruction module 310 may store the variants with theassociated components in the protocol database 390. In some embodiments,the instruction module 310 may determine a set of projections forperforming the protocol using each of the one or more variants. Eachprojection is a list of steps for the protocol using the variant and mayinclude manual (e.g., performed by a human operator) and/or automatic(e.g., performed by a robot 150) steps. The instruction module 310stores the projections for each variant in the protocol database 390.

The instruction module 310 may also determine variants of the operationsof the steps. Variants of the operations are alternate methods ofperforming the operations. For instance, the operation “separating thereagent into 10 holders” may be mapped to the automatic operation of“dividing” a reagent into equivalent amounts between 10 holders. Theinstruction module 310 may determine that a human operator manuallyadding the reagent to each of the ten holders would be a variant of theoperation that is a different way of resulting in the same outcome(e.g., the reagent separated into the holders). The instruction module310 may cross-reference in the lab database to determine variants or mayapply a machine learned model 370 to each operation to determine avariant. The machine learned model 370 may be trained on operationslabeled by a human operator with a set of variants. The instructionmodule 310 may store the variants with the associated operations in theprotocol database 390. In some embodiments, the instruction module 310may store the variants with indication of whether a variant is doneautomatically (e.g., by a robot 150) or manually (e.g., by a humanoperator).

The instruction module 310 may also determine variants of the labs 140for performance of the protocol. Alternate labs 140 (e.g., the variants)include enough of the same or similar components and may have enough ofthe same or similar operations performed in the alternate lab 140 to theselected lab 140 that the protocol can be performed in the alternate lab140. The instruction module 310 may access the variants of thecomponents and operations from the protocol database 390 and input thecomponents and operations, along with their variants, to a machinelearned model 3670 configured to select alternate labs based oncomponents and variants. The machine learned model 370 may be trained onlabs 140 labeled with components that are in and operations that may beperformed in the lab 140. In other embodiments, the instruction module310 cross-references the components, operations, and the variants in thelab database 380 to select a lab 140 that includes components and mayhave operations performed in the lab 140 to complete the protocol. Theinstruction module 310 stores the variants of the labs in associationwith the labs in the lab database 380 and/or the protocol database 390.

The instruction module 310 sends the set of steps, each with anidentified operation, lab equipment 170, robot 150, and/or reagent 180,to the protocol module 330 along with the lab 140 to perform theprotocol in. In some embodiments, the instruction module 310 may sendone or more variants to the graphic user interface module 355 forselection by a user, and upon receiving a selection, send the selectionto the protocol module 330 for performance in the lab 140. In furtherembodiments, the instruction module 310 may receive requests forvariants for a protocol from the protocol module 330 and communicatewith the protocol module 330 to determine the variants for the protocol.The instruction module 310 also sends the set of steps and associatedoperations, lab equipment 170, robots 150, reagents 180, and lab 140 tothe simulation module 340 for simulation in a virtual representation ofthe lab 140. In some embodiments, the instruction module 310 may sendone or more variants to the graphic user interface module 355 forselection by a user, and upon receiving a selection, send the selectionto the simulation module 340 for simulation in the virtualrepresentation.

In some embodiments, the instruction module 310 may group one or moreprotocols for performance in a lab. For instance, the instruction module310 may receive multiple sets of instructions indicating protocols toperform in a lab 140. The instruction module 310 may determine, based onthe components required for each protocol, which protocols can beperformed simultaneously or sequentially in a single lab 140 and createa grouped protocol including steps of the determined protocols. Theinstruction module 310 may send the grouped protocol to the protocolmodule 330 for performance in the lab 140.

The rendering module 320 renders virtual representations of labs 140. Avirtual representation (or graphical representation) of a lab 140includes one or more virtual elements representing components in the lab140 in positions corresponding to the actual positions of the componentsin the actual lab 140. The virtual elements may be labeled based on thecorresponding components. In some embodiments, the virtual elements mayinclude a virtual operator representing a human operator who may performmanual operations in a lab 140. Examples of virtual representations areshown in FIGS. 2A and 2C.

The rendering module 320 receives image data from a camera system 160 ofa lab 140. The image data may depict a lab 140 and include camera andvideo data of the lab 140. In some embodiments, the rendering module 320may also receive, from the graphic user interface module 355, a list ofcomponents and corresponding coordinates in the lab 140. The renderingmodule 320 may also receive sensor data from one or more components inthe lab 140.

The rendering module 320 creates virtual elements representing eachcomponent based on the image data of the component (e.g., if a liquidhandling robot is shown in the image data, the rendering module 320creates a virtual element depicting the liquid handling machine). Insome embodiments, the rendering module 320 saves the virtual elements inthe representation database 360 and uses the virtual elements forsimilar components when rendering other new labs 140. The renderingmodule 320 renders a virtual representation of the lab 140. In someembodiments, the rendering module 320 dynamically localizes a scene ofthe lab 140 from the image data using a three-dimensional model thatperforms spatial abstraction to create the virtual representation. Inother embodiments, the rendering module 320 renders the virtualrepresentation by mapping the virtual elements shown in the image datato a virtual area representing the new lab 140. For instance, therendering module 320 may determine a location of a robot 150 based onthe image data (and/or sensor data indicating a position of the robot)and maps a virtual element representing the robot 150 to a correspondinglocation in the virtual area. The rendering module 320 stores thevirtual rendering in the rendering module 320 in association with thelab 140.

The rendering module 320 may also store information received from thegraphic user interface module 355 about the components in the lab 140 inthe lab database 380 in relation to the lab 140. For instance, therendering module 320 may receive text for labeling the virtual elementfrom the graphic user interface module 355. The rendering module 320 maydetermine a user associated with the test and stores the text inassociation with the virtual element and the user in the representationdatabase 360. The rendering module 320 may additionally store the textin association with the component the virtual element represents in thelab database 380 and/or protocol database 390, such that the user mayreference the component using the text when requesting protocols beperformed in a lab 140. The rendering module 320 may also embed imagedata into the virtual representation. For instance, the rendering module320 may embed image data of components form various angels in the lab140. In another instance the rendering module 320 may embed image datadepicting instructions for performing steps of a protocol or a specificoperation in the lab 140.

The protocol module 330 configures robots 150 in a lab 140 to performprotocols. The protocol module 330 receives a set of steps andassociated robots 150, operations, lab equipment 170, and reagents 180from the instruction module 310. For each step, the protocol module 330configures the associated with the step to perform the operationassociated with the step. The protocol module 330 may additionallyconfigure the robot 150 to interact with the lab equipment 170 (if any)and access and use the reagent 180 (if any) associated with the step toperform the operation. In some embodiments, the protocol module 330 mayrequest variants for the protocol from the instruction module 310 andmodify the protocol with one or more variants. The protocol module 330may, in some instances, request for the simulation module 340 tosimulate the protocol with one or more variants to determine anexperimentation time for the protocol. The experimentation time is theestimated amount of time needed to complete the protocol in the lab 140.The protocol module 330 may select the variants associated with thelowest experimentation time and modify the protocol in the protocoldatabase 390 to use those variants.

The simulation module 340 simulates protocols occurring within labs 140.The simulation module 340 receives request to simulate a protocol in alab 140. The request may be from the graphic user interface module 355.The simulation module 340 may request a set of steps for the protocolfrom the instruction module 310 or access a set of steps for theprotocol from the protocol module 390. In another embodiment, therequest may be in the form of receiving a set of steps to be performedin a lab 140 and associated operations, lab equipment 170, robots 150,and reagents 180 from the instruction module 310.

The simulation module 340 accesses a virtual representation of the lab140 in the representation database 360. For each step received from theinstruction module 310 for the protocol, the simulation module 340determines which virtual elements of the virtual representationcorrespond to the components associated with the step. The simulationmodule 340 determines, based on the operation associated with the step,how the one or more components would need to move within the lab 140 forthe operation to be performed and moves the corresponding virtualelements accordingly in the virtual representation of the lab 140. Thesimulation module 340 may additionally highlight the virtual elements inthe virtual representation. The simulation module 340 accesses the labdatabase 380 to determine whether the movement of components associatedwith the virtual elements would cause any issues during performance of aprotocol. Issues may include robots 150 or lab equipment 170overlapping, a robot 150 being unable to perform an operation, lack ofavailability of lab equipment 170 and reagents 180 in the lab 140, andthe like, and the lab database stores information describing whichcomponents have the ability to stack, overlap, and interact with othercomponents. The simulation module 340 sends the virtual representationwith the moved virtual elements and any detected issues to the graphicuser interface module 355 for each step.

The simulation module 340 also simulates protocols occurring within alab 140 in real-time. The simulation module 340 receives requests fromthe graphic user interface module 355 for simulations of protocols beingcurrently performed in labs 140. For a request, the simulation module340 accesses a virtual representation of a lab 140 in the representationdatabase 360. The simulation module 340 receives sensor data from robots150, equipment 170, and a camera system 160 in a lab 140. The sensordata may include image data of the lab 140, position data of componentsof the robots 150 and/or equipment 170, and any other suitable sensordata. Based on the sensor data, the simulation module 340 determines howthe components within the lab 140 have been moved as a protocol isperformed and moves corresponding virtual elements in the virtualrepresentation of the lab 140 to mirror the movement of the components.The simulation module 340 may additionally highlight virtual elementscorresponding to components being used or moved I the lab 140. Thesimulation module 340 sends an indication to the graphic user interfacemodule 355 that the virtual representation of the lab 140 is beingupdated to show the protocol being performed in real-time. Once theprotocol is completed, the simulation module 340 stores the virtualrepresentation of the lab 140 showing the current state of the lab 140(e.g., the current positions of the components in the real world lab140) in the representation database 360.

The simulation module 340 may also simulate variants of a protocol. Thesimulation module 340 may receive requests from the graphic userinterface module 355 to simulate one or more variants for a protocol. Insome embodiments, the request may be associated with a set ofparameters. The parameters may describe a desired protocol location forperforming the protocol, a desired proximity to the protocol location,equipment preferences, reagent preferences, and the like. The simulationmodule 340 may select one or more variants to simulate for the protocolto optimize satisfying the parameters (e.g., selecting operations thattake less time, cost within the budget, etc.). The simulation module 340modifies a copy of the virtual representation of the lab 140 to depictthe simulation of each variant. In some embodiments, the simulationmodule 340 may modify the copies in different colors depending on thetype of variant. For example, the simulation module 340 may modify thecopies to show variants that occur automatically in a first color andvariants that occur manually in a second color. The simulation module340 may store the modified copies in the representation database 360 inassociation with the variants.

In some embodiments, the simulation module 340 may receive a list ofprojections for a variant and simulate the projections in the lab 140. Aprojection is a list of steps and associated operations and componentsnecessary to perform the entire protocol. The simulation module 340 maydetermine, for each projection, potential errors that may occur if theprotocol were performed according to the projection, such as running outof lab equipment 170 or reagents 180, robots 150 blocking one another inthe lab 140, scheduling conflicts, and the like. The simulation module340 modifies a copy of the virtual representation based on simulation ofthe projections and sends the modified virtual representation to thegraphic user interface module 355 for display after completion of thesimulation of each step.

The calibration module 350 calibrates cameras of the camera systems 160connected to the lab automation system 100. The calibration module 350receives requests to calibrate one or more cameras from the graphic userinterface module 355. In some embodiments, the calibration module 350may periodically calibrate the cameras connected to the lab automationsystem 100 without receiving an explicit request from the graphic userinterface module 355. The calibration module 350 requests image datafrom the one or more cameras and associates subsets of the image datawith its respective camera that captured the image data. The calibrationmodule 350 stores the image data in the lab database 380 in associationwith the lab 140 and the camera.

For each camera of the one or more cameras, the calibration module 350determines which lab 140 the camera is located in and requests sensordata from sensors connected to one or more components in the lab 140.The calibration module 350 stores the sensor data in the lab database380 along with identifiers of the components and the lab 140. Thecalibration module 350 determines a position of one or more elements 220of each component based on the sensor data. For instance, the sensordata may indicate position coordinates of a robot arm 225 and robot hand230 of a robot 150 in the lab 140. In another instance, the positiondata may indicate the position of an element 220 of a robot 150 relativeto a stationary base of the robot 150, the coordinates for which thecalibration module 350 may access in the lab database 380.

The calibration module 350 locates tags physically attached to thecomponents in the image data. Each tag may include information encodedon the tag indicating which element 220 of a component the tag is on andmay vary in shape, size and color. The calibration module 350 may storeinformation describing where the tags are located and shape, size, andcolor of the tags in the lab database 380. The calibration module 350may apply a machine-learned model 370 to the image data to locate tagswithin the image data. The machine-learned model 370 may be trained onimage data with pixels labeled as including a tag or not and coordinatesof the tag and may be a classifier, regression model, decision tree, orany suitable model. The calibration module 350 may further determinedepth information about the image data based on the tags shown in theimages and each tag's associated shape, size, and color.

The calibration module 350 determines the location of the camera basedon the determined positions of the components and locations of tags,such as by triangulating. In some embodiments, the calibration module350 may also account for distortion of the lens of the camera. Thedistortion may have been previously entered for the camera by anexternal operator or the calibration module 350 may determine thedistortion based on the locations of the tags and/or positions of thecomponents. The calibration module 350 calibrates each camera based onthe camera's determined location such that the calibration module 350may determine locations of other components shown in image data capturedby the camera given the camera's determined location.

The calibration module 350 may determine the location of components in alab 140 using image data from calibrated cameras. The calibration module350 may receive a request for the location of a particular component ina lab 140 or may periodically determine locations of components in eachlab 140. The calibration module 350 accesses image data captured by oneor more calibrated cameras in the lab 140 and locates one or more tagsthat are visible on the particular component. The calibration module 350determines the location of the particular component based on thelocation of the one or more tags.

In some embodiments, the calibration module 350 may determine that acamera needs recalibration based on new image data captured by thecamera. The calibration module 350 receives new image data from thecamera in real-time and determines one or more components shown in thenew image data. The calibration module 350 may also request new sensordata from the one or more components (if available, such as when one ormore of the components is a robot 150 with an internal camera) in thenew image data. The calibration module 350 retrieves historical imagedata and corresponding historical sensor data from the lab database 380captured by the camera and determines which components have not moved inthe lab 140 based on the historical sensor data and the new sensor data.The calibration module 350 compares the location of the components thathave not moved between the historical image data and the new image data.In some embodiments, the calibration module 350 may do so using amachine-learned model 370 trained on sets of image data labeled withdiscrepancies (e.g., a component appearing in unexpected pixels in theimage data). If the calibration module 350 determines that a componentdoes not appear where expected, the calibration module 350 recalibratesthe camera. Alternatively, the calibration module 350 may determinewhich pixels of the image data should show the component in an expectedlocation based on the new sensor data and analyze the new image data todetermine if the component is in the expected location. The calibrationmodule 350 recalibrates the camera if the component is not shown in thedetermined pixels.

The graphic user interface module 355 generates graphic user interfacesfor display on one or more client devices 120 connected to the labautomation system 100. Examples of graphic user interfaces are describedwith respect to in FIGS. 4-7. The graphic user interface module 355receives, from a client device 120, a request to view a virtualrepresentation of a lab 140. The graphic user interface module 355retrieves the virtual representation from the representation database360 or requests a virtual representation from the rendering module 320.The graphic user interface module 355 renders the virtual representationin the graphic user interface. The virtual elements of the virtualrepresentation may be interactive elements that a user may interactwith. For instance, upon receiving a mouseover of a virtual element viathe graphic user interface, the graphic user interface module 355 maycause the virtual element to become highlighted within the virtualrepresentation or mimic an operation being performed. Further, uponselection of a virtual element, the graphic user interface module 355may present a tag element connected to the virtual element. A user mayenter text to the tag element to label the virtual element and itsassociated component with. The graphic user interface module 355 sendsthe text to the rending module 320 for addition to the virtualrepresentation in the representation database 360.

The graphic user interface module 355 renders one or more interactiveelements in the graphic user interface. The interactive elements mayallow a user to move virtual elements associated with components in alab, enter coordinates of components in a lab, request a simulation of aprotocol, request calibration of one or more cameras, request variantsfor a protocol, and the like. The interactive elements may also allow auser to select a mode for the graphic user interface to operate in. Forexample, a first mode may cause the graphic user interface to displaysimulations of protocols in a virtual presentation and a second mode maycause the graphic user interface to mimic, in real-time, protocol thatis currently being performed in a lab.

The graphic user interface module 355 may display one or moresimulations of protocols in a lab via the graphic user interface. Thegraphic user interface module 355 receives, via an interaction with aninteractive element of the graphic user interface, a request to simulatea protocol in a lab 140. The graphic user interface module 355 requestsa simulation of the protocol in the lab from the simulation module 340.The graphic user interface module 355 receives a virtual representationof the lab with virtual elements moved for each step of a protocol andpresents the virtual representation via the graphic user interface. Insome embodiments, the graphic user interface may include an interactivescrolling element that allows a user to increment through each step inthe virtual representation. The graphic user interface module 355 mayreceive detected issues for each step from the simulation module 340 andpresents the detected issues as alerts via the graphic user interface.The detected issues to the graphic user interface module 355 for eachstep.

In some embodiments, the request for the simulation may include one ormore parameters. Examples of parameters include a desiredexperimentation time for the protocol, a budget, a necessary skillsetfor human operators, necessary equipment 170 and/or reagents 180, andthe like. The graphic user interface module 355 sends the request withthe parameters to the simulation module 340, which determines one ormore variants to simulate to satisfy the parameters. Upon displaying thesimulation in a virtual representation, the graphic user interfacemodule 355 may display statistics representing whether the parametersare satisfied or not using the one or more variants.

The graphic user interface module 355 may also receive, via aninteraction with an interactive element of the graphic user interface, arequest to simulate a protocol that is currently occurring in a lab 140.The graphic user interface module 355 requests a simulation of theprotocol the simulation module 340. The graphic user interface module355 receives indications from the simulation module 340 as thesimulation module 340 updates the virtual representation of the lab 140in the representation database 360. The graphic user interface module355 displays the updated virtual representation in real-time via thegraphic user interface. In some embodiments, the graphic user interfacemodule may also send notifications in response to receiving a request.For instance, if the graphic user interface module 3555 receives arequest for variants of a robot 150 in a lab 140 but the instructionmodule 310 indicates that none exist, the graphic user interface module355 displays a notification on the graphic user interface indicatingthat no variants are available for the robot 150.

The graphic user interface module 355 may receive instructions via atext box of the graphic user interface. The instructions may beassociated with a lab 140 and, in some embodiments, a set of parameters.The graphic user interface module 355 sends the instructions with thelab 140 and parameters to the instruction module 310. The graphic userinterface module 355 may receive an indication of an ambiguity or errorin the instructions from the instruction module 310. The graphic userinterface module 355 may present one or more words associated with anambiguity via the graphic user interface and receive a selection of oneor more words via the graphic user interface, which the graphic userinterface module 355 sends to the instruction module 310. The graphicuser interface module 355 may also present one or more componentsassociated with an error via the graphic user interface and receive aselection of one or more components via the graphic user interface,which the graphic user interface module 355 sends to the instructionmodule 310.

The graphic user interface module 355 may receive one or more variantsfrom the instruction module 310 and displays the one or more variants ina selectable list via the user interface. For example, the variants maybe a list of alternate labs 140 for the lab 140 for a user to selectfrom. The graphic user interface may highlight, in the virtualrepresentation, virtual elements of components associated with a variantupon receiving mouseover of the variant in the selectable list. Inanother embodiment, the graphic user interface module 355 may highlightvirtual elements representing the variants in the virtual representationsuch that a user may interact with a highlighted virtual element toselect a variant. The graphic user interface module 355 sends selectedvariants to the instruction module 310.

Graphic User Interface Examples

FIG. 4 illustrates a graphic user interface for the lab automationsystem 100, according to one embodiment. In other examples, the graphicuser interface 400 may include additional, fewer, or different elementsthan those shown in FIG. 4. The graphic user interface 400 may begenerated by the graphic user interface module 355 described previously.In the example of FIG. 4, the graphic user interface 400 comprises afirst interface portion 410 and a second interface portion 420. Thefirst interface portion 410 enables a user of the lab automation system100 to provide instructions in natural language to the lab automationsystem 100, which identifies operations, equipment, and/or reagentscorresponding to the provided instructions, as described in reference tothe instruction module 310. In some embodiments, the first interfaceportion 410 additionally enables a user to simulate or execute aprotocol responsive to steps being generated for provided instructions.

The second interface portion 420 comprises a view of a lab 140. In someembodiments, the second interface portion 420 is a virtualrepresentation of the lab 140 generated by the rendering module 320.Users of the graphic user interface 400 may modify the virtualrepresentation by interacting with virtual elements 430 via the graphicuser interface 400 and may request simulations of robots 150 performingthe steps of a protocol for visualization or display within the virtualrepresentation. In other embodiments, the second interface portion 420is a live feed, representation, or view of the lab 140 captured via acamera system 160. In these embodiments, the second interface portion420 displays the lab 140 in real-time or in near real-time as robots 150perform protocols.

FIG. 5A illustrates an example graphic user interface 500A for inputtinginstructions, according to one embodiment. In particular, the firstinterface portion 410 of the graphic user interface 500A includes a textbox 510 and a predictive text interface element 520, and the secondportion 420 includes a virtual representation 515 of a lab 140. A usermay enter instructions to the text box 510, which are sent by thegraphic user interface module 355 to the instruction module 310. Thegraphic user interface module 355 receives one or more pieces ofequipment 170 (or other components) determined by the instruction module310 to correspond to the ambiguous term 530. The one or more pieces ofequipment are listed in the predictive text interface element 520 suchthat upon mouseover 550 of a virtual element 540 representing a piece ofequipment 170 in the predictive text interface element 530, the graphicuser interface module 355 highlights the virtual element 540A in thevirtual representation 515.

Upon selection of one of the pieces of equipment 170 listed in thepredictive text interface element 520, the graphic user interface module355 modifies a parsed version of the instructions 570 to include theselected piece of equipment 170, as shown in FIG. 5B. The graphic userinterface module 355 displays the parsed version of the instructions 570on the graphic user interface 500B. Upon mouseover of the selected pieceof equipment 560, the graphic user interface module 355 may highlight540B or add a virtual element 540B representing the selected piece ofequipment 560 in the virtual representation 515.

FIG. 6A illustrates an example graphic user interface 600A depicting aselection of a virtual element 605A within a virtual representation 615of a lab 140, according to one embodiment. In particular, a user maymouseover virtual elements 605 representing pieces of equipment 170shown in the virtual representation 615 of the graphic user interface600A and select 610 a virtual element of a piece of equipment 170 tolabel. As shown in FIG. 6B, upon selection of a piece of equipment, thegraphic user interface module 355 connects the virtual element 605Arepresenting the selected piece of equipment 170 with a tag element 620in the graphic user interface 600A. A user may enter text for labelingthe virtual element 605A via the tag element 620. The graphic userinterface module 355 sends the text entered to the tag element 620 tothe rendering module 320, which updates the virtual representation 615of the lab 140 to include the virtual element labeled with the text. Forexample, if a user labels the virtual element as “chemical box,” theinstruction module 310 may recognize instructions including a “chemicalbox” as corresponding to the piece of equipment represented by thevirtual element 605A.

FIG. 7 illustrates an example graphic user interface 700 depicting asimulation, according to one embodiment. A user may request a simulationof a protocol in a lab 140 via interactive elements 715 of the graphicuser interface 700. The graphic user interface module 355 receivesvirtual representations 705 from the simulation module 340 for each stepof a protocol and modifies the graphic user interface 700 to display thevirtual representations for each step in the order dictated by theprotocol. The graphic user interface 700 may additionally showsimulation information about the simulation, such as statisticsrepresenting the likelihood of success of the protocol being performedin the lab 140, a time to competition, a time the simulation has beenrunning, which step the simulation is currently running, which stepshave already been run for the simulation, and any other suitableinformation about the simulation.

Processes

FIG. 8 illustrates a process 800 for configuring a robot 150 to performsteps for a protocol, according to one embodiment. In particular, thegraphic user interface module 355 receives 805, via a graphic userinterface, an instruction from a user to perform a protocol within a lab140. The instruction may comprise text indicating the instructions andthe lab 140 to perform the protocol associated with the instructions in.The graphic user interface module 355 sends the instructions to theinstruction module 310, which converts 810, using a machine learnedmodel 370, the text into steps. Each step may be a segment of the textindicating an action to be performed for the protocol. In someembodiments, the machine learned model 370 may use natural languageprocessing to extract generic capabilities (e.g., actions) from thetext.

For each step, the instruction module 310 identifies 815 one or more ofan operation such as the action), lab equipment 170, and reagent 180associated with the step. For instance, the instruction module 310 mayaccess the lab database 380 to determine which components are availablein the lab 140 and select only available lab equipment 170 and reagents180 for the step. In response to detecting an ambiguity or errorassociated with the step, the instruction module 310 alerts the graphicuser interface module 355, and the graphic user interface module 355notifies 820 the user via the graphic user interface of the ambiguity orerror. For each step, the protocol module 330 configures 825 a robot 150to perform 830 an identified operation, interact 835 with identified labequipment 170, and/or access and use 840 an identified reagent 180associated with the step.

It is appreciated that although FIG. 8 illustrates a number ofinteractions according to one embodiment, the precise interactionsand/or order of interactions may vary in different embodiments. Forexample, in some embodiments, the instruction module 310 may determine adet of candidate actions for the ambiguity or error and select acandidate action with more than a threshold likelihood of resolving theambiguity or error. The instruction module 310 may send the candidateoperation to the graphic user interface module 355, which displays anindication of the step associated with the ambiguity or error via thegraphic user interface. The graphic user interface module 355 may alsodisplay, in a virtual representation of the lab 140, a representativeaction associated with the candidate operation, such that the user cansee what the candidate operation would look like being performed in thelab 140. In other embodiments, the instruction module 310 may determinea list of one or more candidate operations, lab equipment 170, and/orreagents 180 that may resolve the ambiguity or error, and the graphicuser interface module 355 may display the list via the graphic userinterface.

FIG. 9 illustrates a process 900 for simulating a protocol in a virtualrepresentation of a lab 140, according to one embodiment. The simulationmodule 340 identifies 905 a set of steps associated with a protocol fora lab 140. Each step may be meant to be performed by a robot 150 withinthe lab 140 using one or more of lab equipment 170 and reagents 180. Therendering module 320 renders 910 a virtual representation of the lab140, a virtual robot within the lab 140, and virtual equipment andreagents 180 within the lab 140, and the graphic user interface module355 may present the virtual representation in a graphic user interface.The rendering module 320 stores the virtual representation of the lab140 in the representation database 360 such that the simulation module340 may access the virtual representation of the lab 140 to runsimulations.

In response to operating in a first mode, the simulation module 340simulates 920 the identified set of steps being performed by the virtualrobot to identify virtual positions of the virtual robot within the lab140 as the virtual robot performs the identified set of steps andmodifies the virtual representation of the lab 140 to mirror theidentified positions of the virtual robot as the virtual robot performsthe identified set of steps. The graphic user interface module 355 maypresent, via the graphic user interface, the modified virtualrepresentation for each step. In response to operating in a second mode,the simulation module 340 identifies 930 positions of the robot 150within the lab 140 as the robot 150 performs the identified set of stepsand modifies the virtual representation of the lab 140 to mirror theidentified positions of the robot 150 as the robot 150 performs theidentified set of steps. The graphic user interface module 355 maypresent, via the graphic user interface, the modified virtualrepresentation for each step as the step is performed in real-time inthe lab 140.

It is appreciated that although FIG. 9 illustrates a number ofinteractions according to one embodiment, the precise interactionsand/or order of interactions may vary in different embodiments. Forexample, in some embodiments, the instruction module 310 may determineone or more alternate robots 150 for the robot 150 of the protocol,where each of the alternate robots 150 has capabilities that map tocapabilities of the robot 150 (e.g., the alternate robots 150 have theability to perform the same operations as the robot 150). The simulationmodule 340 may simulate the identified set of steps being performed byone or more of the alternate robots 150 and the graphic user interfacemodule 355 may present the simulated steps in the virtual representationof the lab 140 to the user, such that the user may view how the protocolwould be performed using the one or more alternate robots 150.

In some embodiments, the instruction module 310 may determine a set ofvariants for one or more of the robot 150, lab 140, lab equipment 170,and reagents 180. The set of variants may include at least two subsetsof variants, and the first subset may include variants with the samefunctionality as the identified set of steps, robot 150, lab 140, labequipment 170, and reagents 180. For instance, the instruction module310 may determine that the lab equipment 170 “Liquefier 25 ml pipettes”may be replaced by “Science & Co. 25 ml pipettes.” The second subset mayinclude variants with alternate functionality to the identified set ofsteps, robot 150, lab 140, lab equipment 170, and reagents 180 thatstill accomplishes the same result as the identified set of steps, robot150, lab 140, lab equipment 170, and reagents 180. For example, a firstlab 140 may usually be used to perform research in, and a second lab mayusually be used to synthesize carbon nanotubes. The instruction module310 may determine that the labs include enough equipment 170 overlapthat the labs 140 are variants of one another. The simulation module 340may simulate protocols using each of the subset of variants and modifythe virtual representation of the lab 140 for the simulation indifferent colors for each subset (e.g., the first subset may be shown inblue and the second subset may be shown in green). Further, if theinstruction module 310 determines that no variants exist for theidentified set of steps, robot 150, lab 140, lab equipment 170, andreagents 180, the instruction module 310 sends an indication to thegraphic user interface module 355 to notify a user that no variantsexist.

In some embodiments, the instruction module 310 may determine multiplesets of variants for a protocol. For instance, a user may request, viathe graphic user interface, to see simulations of a protocol beingperformed partially manually and fully automated. The instruction module310 may select two sets of variants for the protocol. The first set mayinclude a mix of variants related to both automatic and manualoperations of the protocol, and the second set may only include variantsrelated to automatic operations of the protocol. The instruction modulesends the sets of variants for the protocol to the simulation module340, which simulates the protocol being performed with each set ofvariants. The graphic user interface module 355 may display thesimulations via the graphic user interface such that a user can see thedifferences between performing the protocol semi manually and fullyautomatically.

FIG. 10 illustrates a process 1000 for modifying a virtualrepresentation, according to one embodiment. The instruction module 310identifies 1005 a set of steps associated with a protocol for a lab 140,each step being associated with lab equipment 170 and reagents 180. Theinstruction module 310 identifies 1010 a plurality of protocol variants.Each of the protocol variants is associated with a first subset of stepsperformed by a robot 150 and a second subset of steps performed by ahuman operator. The rendering module 320 renders 1015 a virtualrepresentation of the lab 140, which the graphic user interface module355 may display via a graphic user interface. The simulation module 340simulates 1020 each protocol variant by modifying the virtualrepresentation to show the first subset of steps being performed by avirtual robot and the second subset of steps being performed by avirtual human operator. The simulation module 340 modifies 1025 thevirtual representation to include characteristics of each protocolvariant. The characteristics may include one or more of an amount oftime each simulated protocol variant requires, a resource requirement ofthe simulated protocol variant, and a measure of automation of thesimulated protocol variant.

It is appreciated that although FIG. 10 illustrates a number ofinteractions according to one embodiment, the precise interactionsand/or order of interactions may vary in different embodiments. Forexample, in some embodiments, the characteristics may further includestatistics describing expected delays from human operators performingthe second subset of steps in the lab 140 and potential schedulingconflicts with the components of the lab 140.

FIG. 11 illustrates a process 1100 for determining a location of labequipment 170 relative to a robot arm 225, according to one embodiment.The calibration module 350 accesses 1105, via a camera within a lab 140,a first image of a robot arm 225 (or another component of a robot 150 orlab equipment 170 in the lab 140). The robot arm 225 (or othercomponent) comprises a visible tag located on an exterior. Thecalibration module 350 determines 1110 a position of the robot arm 225using one or more position sensors located within the arm and determines1115 a location of the camera relative to the robot arm based on thedetermined position and the location of the visible tag in the firstimage (e.g., by triangulating). The calibration module 350 calibrates1120 the camera using the determined location of the camera relative tothe robot arm 225. After calibrating the camera, the calibration module350 accesses 1130, via the camera, a second image of equipment 170 (or,in some embodiments, another robot 150) that comprises a second visibletag on an exterior. The calibration module 350 determines 1135, based ona location of the second visible tag within the accessed second image, alocation of the equipment 170 relative to the robot arm 225.

It is appreciated that although FIG. 11 illustrates a number ofinteractions according to one embodiment, the precise interactionsand/or order of interactions may vary in different embodiments. Forexample, in some embodiments, the calibration module 340 may access, viathe camera, a third image of the robot arm 225, where the robot arm 225is in a different position than in the first image. The calibrationmodule may determine that recalibration of the camera is necessary basedon the third image and corresponding position data of the robot arm 225.The calibration module 350 determines a second location of the camerarelative to the robot arm 225 based on a second determined position ofthe robot arm 225 in the third image and a second location of the tagwithin the third image. The calibration module 350 calibrates the camerausing the second location of the camera.

FIG. 12 illustrates a process 1200 for configuring a second robot 150,according to one embodiment. The protocol module 330 accesses 1205, fromthe protocol database 390, a first protocol to be performed by a firstrobot 150 in a first lab 140. The first protocol includes a set ofsteps, each associated with one or more of an operation, equipment 170,and a reagent 180 in the first lab 140. For each of one or more of theset of steps, the protocol module 330 modifies 1210 the step by one ormore of: (1) identifying 1215 one or more replacement operations (e.g.,variants) that, when performed, achieve an equivalent or substantiallysimilar result as a performance of the operation associated with thestep, (2) identifying 1220 replacement equipment 170 (e.g., variants)that operate substantially similarly to the equipment 170 associatedwith the step, and (3) identifying 1225 one or more replacement reagents180 (e.g., variants) that, when substituted for the reagent 180associated with the step, do not substantially affect the performance ofthe step. In some embodiments, the graphic user interface module 355 mayreceive a selection of a virtual element representing equipment 170 inthe lab 140 and send an indication to the protocol module 330 to use theequipment 170 in the modified protocol.

The protocol module 330 generates 1230 a modified protocol by replacingone or more of the set of steps from the first protocol with the one ormore modified steps. The protocol module 330 may request a set ofcandidate labs labs from the instruction module 310, which determinesthe set of candidate labs for the modified lab protocol. Each of thecandidate labs 140 includes the identified replacement equipment 170 andreplacement reagents 180. The instruction module 310 selects 1235, froma set of candidate labs 140, a second lab 140 including a second robot150 that can perform the modified protocol. In some embodiments, theinstruction module 310 may send the set of candidate labs to the graphicuser interface module 355 for display to a user in a selectable list andreceives a selection of a candidate lab to use as the second lab. Theinstruction module 310 sends the selection to the protocol module 330,which modifies the protocol to include the second lab 140. The protocolmodule 330 configures 1240 the second robot 150 to perform the modifiedprotocol in the second lab 140.

It is appreciated that although FIG. 12 illustrates a number ofinteractions according to one embodiment, the precise interactionsand/or order of interactions may vary in different embodiments. Forexample, in some embodiments, the instruction module 310 may request,from the simulation module 340, an experimentation time for performingthe modified lab protocol in each of the set of candidate labs 140. Thesimulation module 340 may run a simulation of the modified lab protocolto determine the experimentation time for each candidate lab 140 basedon the equipment 170 and reagents 180 available in the candidate lab 140and the schedule of the candidate lab 140. The instruction module 310may select the second lab from the candidate labs based on thedetermined experimentation times received from the simulation module340.

FIG. 13 illustrates a process 1300 for modifying an instruction uponselection of a piece of equipment 170, according to one embodiment. Theinstruction module 310 receives 1305, from the graphic user interfacemodule 355, an instruction to perform a protocol within a lab 140. Theinstruction may be entered by a user via a graphic user interfaceincluding a virtual representation of the lab 140. The lab 140 includesa robot 150 and set of equipment 170 rendered within the virtualrepresentation of the lab 140. The instruction module 310 identifies1310 an ambiguous term of the instruction and one or more pieces ofequipment 170 corresponding to the ambiguous term and modifies theinterface to include a predictive text interface element listing the oneor more pieces of equipment 170 corresponding to the ambiguous term.Upon a mouseover of a listed piece of equipment 170 within thepredictive text interface element, the graphic user interface module 355modifies 1320 the virtual representation of the lab 140 to highlight thelisted piece of equipment 170 corresponding to the mouseover. In someembodiments, the graphic user interface module 355 may further modifythe virtual representation to depict a virtual element representing thelisted piece of equipment performing an operation within the lab 140.Upon a selection of the listed piece of equipment 170 within thepredictive text interface element, the graphic user interface module 355modifies 1330 the instruction to include the listed piece of equipment170.

It is appreciated that although FIG. 13 illustrates a number ofinteractions according to one embodiment, the precise interactionsand/or order of interactions may vary in different embodiments. Forexample, in some embodiments, the graphic user interface module 355 mayreceive text representative of a name of the listed piece of equipment150 along with the selection of the listed piece of equipment 150. Thegraphic user interface module 355 may update the virtual representationto include the text for the piece of equipment in the representationdatabase 360 or update the name of the piece of equipment 150 to matchthe text in the lab database 380.

In some embodiments, the graphic user interface module 355 mayadditionally display, via the graphic user interface, a list ofprojections for performing the protocol using the listed piece ofequipment 150. Each projection comprises a list of manual and automaticsteps required for the protocol and may be associated with a time frameindicating an amount of time required to perform the protocol using thesteps from the projection. Upon selection of a projection, the graphicuser interface module 355 requests a simulation of the protocol beingperformed using the operation of the projection in the lab from thesimulation module 340 and displays the simulation according to theprojection at the graphic user interface. The graphic user interfacemodule 355 may additionally display one or more alerts indicatingpotential errors that may occur when the protocol is performed accordingto the projection.

Summary

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: receiving, by a labautomation system, an instruction from a user to perform a protocolwithin a lab via an interface including a graphical representation ofthe lab, the lab including a robot and set of lab equipment renderedwithin the graphical representation of the lab; identifying, by the labautomation system, an ambiguous term of the instruction and one or morepieces of equipment corresponding to the ambiguous term; modifying, bythe lab automation system, the interface to include a predictive textinterface element listing the one or more pieces of equipmentcorresponding to the ambiguous term; upon a mouseover of a listed pieceof equipment within the predictive text interface element, modifying, bythe lab automation system, the graphical representation of the lab tohighlight the listed piece of equipment corresponding to the mouseover;and upon a selection of the listed piece of equipment within thepredictive text interface element, modifying, by the lab automationsystem, the instruction to include the listed piece of equipment.
 2. Themethod of claim 1, further comprising: receiving, with the selection ofthe listed piece of equipment, text representative of a name of thelisted piece of equipment; and updating a lab database to label thelisted piece of equipment with the text.
 3. The method of claim 1,wherein modifying the graphical representation of the lab to highlightthe listed piece of equipment corresponding to the mouseover furthercomprises: modifying, by the lab automation system, the graphicalrepresentation of the lab to depict the listed piece of equipmentperforming an operation in the lab.
 4. The method of claim 1, furthercomprising: displaying, via the interface, a list of projections forperforming the protocol using the selected listed piece of equipment;upon selection of a projection of the list, displaying, via thegraphical representation of the lab, a simulation of the listed piece ofequipment performing the protocol according to the projection.
 5. Themethod of claim 4, wherein the projections in the list comprise acombination of automatic and manual steps for the protocol.
 6. Themethod of claim 4, wherein each projection is associated with a timeframe of an amount of time required to perform the protocol according tothe projection.
 7. The method of claim 4, further comprising:displaying, via the interface, one or more alerts indicating potentialerrors that may occur when the protocol is performed according to theprojection.
 8. A non-transitory computer-readable storage medium withinstructions executable by a processor comprising: instructions forreceiving, by a lab automation system, an instruction from a user toperform a protocol within a lab via an interface including a graphicalrepresentation of the lab, the lab including a robot and set of labequipment rendered within the graphical representation of the lab;instructions for identifying, by the lab automation system, an ambiguousterm of the instruction and one or more pieces of equipmentcorresponding to the ambiguous term; instructions for modifying, by thelab automation system, the interface to include a predictive textinterface element listing the one or more pieces of equipmentcorresponding to the ambiguous term; instructions for, upon a mouseoverof a listed piece of equipment within the predictive text interfaceelement, modifying, by the lab automation system, the graphicalrepresentation of the lab to highlight the listed piece of equipmentcorresponding to the mouseover; and instructions for, upon a selectionof the listed piece of equipment within the predictive text interfaceelement, modifying, by the lab automation system, the instruction toinclude the listed piece of equipment.
 9. The non-transitorycomputer-readable storage medium of claim 8, the instructions furthercomprising: receiving, with the selection of the listed piece ofequipment, text representative of a name of the listed piece ofequipment; and updating a lab database to label the listed piece ofequipment with the text.
 10. The non-transitory computer-readablestorage medium of claim 8, wherein the instructions for modifying thegraphical representation of the lab to highlight the listed piece ofequipment corresponding to the mouseover further comprise: instructionsfor modifying, by the lab automation system, the graphicalrepresentation of the lab to depict the listed piece of equipmentperforming an operation in the lab.
 11. The non-transitorycomputer-readable storage medium of claim 8, wherein the instructionsfurther comprise: instructions for displaying, via the interface, a listof projections for performing the protocol using the selected listedpiece of equipment; instructions for, upon selection of a projection ofthe list, displaying, via the graphical representation of the lab, asimulation of the listed piece of equipment performing the protocolaccording to the projection.
 12. The non-transitory computer-readablestorage medium of claim 11, wherein the projections in the list comprisea combination of automatic and manual steps for the protocol.
 13. Thenon-transitory computer-readable storage medium of claim 11, whereineach projection is associated with a time frame of an amount of timerequired to perform the protocol according to the projection.
 14. Thenon-transitory computer-readable storage medium of claim 11, wherein theinstructions further comprise: instructions for displaying, via theinterface, one or more alerts indicating potential errors that may occurwhen the protocol is performed according to the projection.
 15. Acomputer system comprising: a processor; and a non-transitorycomputer-readable storage medium storing instructions that, whenexecuted, cause the processor to: receive, by a lab automation system,an instruction from a user to perform a protocol within a lab via aninterface including a graphical representation of the lab, the labincluding a robot and set of lab equipment rendered within the graphicalrepresentation of the lab; identify, by the lab automation system, anambiguous term of the instruction and one or more pieces of equipmentcorresponding to the ambiguous term; modify, by the lab automationsystem, the interface to include a predictive text interface elementlisting the one or more pieces of equipment corresponding to theambiguous term; upon a mouseover of a listed piece of equipment withinthe predictive text interface element, modify, by the lab automationsystem, the graphical representation of the lab to highlight the listedpiece of equipment corresponding to the mouseover; and upon a selectionof the listed piece of equipment within the predictive text interfaceelement, modify, by the lab automation system, the instruction toinclude the listed piece of equipment.
 16. The computer system of claim15, the instructions further causing the processor to: receive, with theselection of the listed piece of equipment, text representative of aname of the listed piece of equipment; and update a lab database tolabel the listed piece of equipment with the text.
 17. The computersystem of claim 15, wherein modifying the graphical representation ofthe lab to highlight the listed piece of equipment corresponding to themouseover further comprises: modify, by the lab automation system, thegraphical representation of the lab to depict the listed piece ofequipment performing an operation in the lab.
 18. The computer system ofclaim 15, the instructions further causing the processor to: display,via the interface, a list of projections for performing the protocolusing the selected listed piece of equipment; upon selection of aprojection of the list, display, via the graphical representation of thelab, a simulation of the listed piece of equipment performing theprotocol according to the projection.
 19. The computer system of claim18, wherein the projections in the list comprise a combination ofautomatic and manual steps for the protocol.
 20. The computer system ofclaim 18, wherein each projection is associated with a time frame of anamount of time required to perform the protocol according to theprojection.